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Abstract 



Operating within a milieu of social, political, and economic 
change, educators are under increasing pressure to provide 
information about their instructional efforts. As instructors are 
called upon to do different types of teaching and to conduct their 
teaching differently, evaluation procedures must also address these 
changes. 

Self -appraisal offers significant opportunity to address 
environmental changes in post- secondary institutes. Teaching 
dossiers are a method of self-evaluation which offer a mechanism 
for educators to move beyond static conceptions of effective 
teaching to reflective and rehable models of performance 
evaluation. 




Multi-faceted Performance Evaluation: 

The Role of Teaching Dossiers 

Provincial funding cuts to post-secondary institutions are leading to 
overcrowded classes at the University of Alberta... Some courses are 
only offered every second year because there are not enough professors 
to teach them. The quality of instruction is decreasing in some fields 
because of old lab equipment or lack of supervisors for graduate 
students ... those professors left are scrambling to teach more classes a 
week that are larger in size because there's no money to replace 
colleagues who have left. (Moysa, 1996) 

Media articles such as the excerpt above reflect the impact of political and 
economic decisions on the role of instructors within post-secondary environments. In 
addition to financial pressures, universities are also faced with an unprecedented 
diversity of students whose different cultural and academic backgrounds, needs, and 
aspirations call for continual improvements in teaching methods (Greene, 1994). 

Operating within this milieu, instructors are under increasing pressure to 
provide information about their instructional efforts. Outside forces are demanding that 
educators be more responsive to the changing teaching environment and institutes to 
rethink the monitoring and measuring of instructional performance within dynamic 
environments. As instructors are called upon to do different types of teaching and to 
conduct their teaching differently, evaluation criteria and procedures must establish 
timely, accurate, and usable systems of performance assessment in post-secondary 
environments. 

The purpose of this paper is two-fold. First, it describes performance evaluation 
within education environments, focusing on those at post-secondary levels. It attempts 
to identify important issues and problems that need to be addressed in order to meet 
rapidly changing needs and expectations of instructors, administrators, students, and 
other stakeholders. From this exploration, a key principle is developed which suggests 
that instructional effectiveness can best be evaluated if it is assessed from a variety of 



perspectives relating to purpose, criteria, source, and method, and if it has regard for 
particular contexts. The paper then focuses on self-evaluation as method of assessing 
instructor performance and, within that context, presents the argument that teaching 
dossiers meet personal and institutional needs for rich evaluative data relating to 
instructional performance in dynamic teaching and learning environments. 

A multi-faceted approach to evaluation serves as the conceptual framework for 
thi s paper. The operating principles of the framework are based on assumptions of 
previous researchers (Braskamp, Brandenburg, and Ory, 1984; Genova, Madoff, Chin & 
Thomas, 1976) who suggested that effective evaluation is multi-purpose, multi-sourced, 
multi-method, and has regard for institutional context relating to particular needs and 
stages of development. A model has been developed (Appendix 1) which conceptualizes 
the complexity of this multi-faceted evaluation process. 

Evaluation: A Multi-faceted Concept 

Stufflebeam and Webster (1988) defined evaluation as a conceptual activity in 

which 



Evaluators and cUents must conceptuahze the evaluation questions, 
information needed to address them, appropriate values and criteria for 
examining and interpeting the information, ways to obtain and analyze the 
information, the structure of reports for communicating findings, and 
appropriate ways to use the findings. They must also conceptualize the 
ways in which evaluation fits into the structure of the school system and 
community: that is, how evaluation is to be governed, organized, 
administered, financed, controlled for bias, and employed within a political 
environment. And they need to develop a shared conception of what 
evaluation means, what it is used for, how it is effectively implemented, 
and how it is properly appraised, (p. 570) 

This description provides an introduction to the complexity of evaluation as a multi- 
faceted activity which attends to environmental constructs. 

While evaluation serves a variety of purposes including analyzing programs and 
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developing policy, a major purpose within organizations is that of appraising employees' 
work activity, more commonly referred to as performance evaluation. Within this 
context, evaluation has been defined more specifically as a process designed "to give 
employees feedback on performance, to identify the employees' developmental needs, to 
make promotion and reward decisions, to make demotion and termination decisions, and 
to develop information about the organization's selection and placement decisions" 
(Nelson & Q.uick, 1994, p. 183). 

While performance evaluation can be understood as the gathering and measuring 
of work performance information as a basis for personal and/or institutional decisions 
(Braskamp & Ory, 1994; Nelson & Quick, 1994), it must be understood as an activity 
which affects not only individuals, but organizations as well. Braskamp and Ory ( 1994) 
employed a "sitting beside" (p. 13-14) metaphor to illustrate the fact that evaluation 
involves judgements and often comparisons either between people or against established 
standards within the organization. For example, an instructor evaluation which 
considers student outcomes considers not just an individual's teaching practices, but 
student results as compared with those of other instructors within the institute and with 
the goals of the institute. 

Genova, Madoff, Chin, and Thomas (1976) elaborated on the significance of 

evaluation to both individuals and institutes: 

Evaluating people is a serious business. At stake for the persons evaluated 
are self-esteem, job security, assignments and promotion, and future 
careers. For students, the quality and usefulness of their education 
depends in large part on the institution's capacity to identify and main- 
tain staff excellence. For the institution itself, its very survival will more 
and more depend on its purposefulness and quality of instruction, (p. 2) 

Performance Evaluation in Education: A Multi-faceted Model 

Multiple purposes. A review of literature addressing performance evaluation 
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in education suggests it serves a number of purposes wi thin that context. Cullen (1995), 
in examining the evaluation of school superintendents, related specific types of 
evaluation to the purpose for which it is intended. Among the purposes she described 
are improving educational performance through instruction, improving communication 
and relations between various levels of administration, clarifying roles, improving 
planning, aiding in professional development and personnel decisions, serving as an 
accountabhity mechanism, and fuhilhng legal requirements. 

The Canadian Association for University Teachers (CAUT) (1991) outhned 
reasons for evaluating teaching as improving the quahty of teaching, helping students 
make choices among courses, and including teaching effectiveness among the criteria for 
career advancement. Because CAUT viewed the improvement of instruction as most 
important to teaching, it suggested data must be collected frequently, results must be 
confidential to the institution, and assistance must be made available to instructors to 
enable them to modify teaching strategies. 

Other researchers (Braskamp & Ory, 1994; Roe & McDonald, 1984) suggested that 
at post-secondaiy levels evaluation can be broadly classified as having dual roles relating 
to describing performance and to judging performance or, in other words, relating to 
individual improvement and to institutional accountability. Under these broad 
categories, a mmiber of major features are itemized as teaching effectiveness, salary 
adjustment, promotion, tenure, curriculum development, and student information. 

These two roles or categories of evaluation have also been termed formative and 
summative evaluation which are explored in the following section. 

Formative and summative evaluations. While research suggests there are 
many purposes and types of appraisal, performance evaluation is often grouped in two 
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broad categories of formative and summative evaluation. Formative is the term applied 
to evaluation that seeks to continue the development of or to improve the subject of 
evaluation. Its focus on development allows instructors who receive poor evaluations to 
improve their performance (Cullen, 1995; Stufflebeam & Webster, 1988). Roe and 
McDonald (1984) added further that because formative evaluation leads to suggestions 
for improvement, it is most often interim and not a final appraisal. 

Summative evaluation, on the other hand, is used for formal decision making. 
Summative refers to evaluation that seeks to provide a statement or summation of 
performance used as input to salary, promotion, and tenure decisions (Cullen, 1995; Roe 
& McDonald, 1984; Stufflebeam & Webster, 1988). The distinctions between formative 
and summative evaluations as they relate to purposes outlined in the hterature are 
presented below. 



Table 1. Types of Instructor Performance Evaluation 



Purpose 



Formative 



Summative 



Measure educational performance 
Improve quality of instruction 
Establish performance goals 
Improve communication 



V 

V 

V 



V 



Help students with course selection 
Improve relations 
Inform expectations 
Improve planning 



V 

V 

V 



V 



Aid in professional development 
Influence personnel decisions 
Serve as accountabihty mechanism 
Fulfill legal requirements 



V 



V 

V 

V 



While some researchers (Roe & McDonald, 1983) consider formative evaluation 
the more important at post-secondary levels, a consideration of the multiple purposes of 



evaluation itemized in the table, along with the changing environments in dynamic 
institutes, suggests it is difficult to estabhsh one as more critical than the other. 

Multiple evaluative criteria. A number of performance criteria have been 
identified in evaluating educators. Johnson and Holdaway (1991), in their study of 
performance indicators of senior high school principals, identified 29 dimensions of 
effective performance from which "a pervasive attitude of 'humanitarian' concern for 
staff and students" (p. 60) emerged as most important among the criteria. 

In another study of school principal performance, Heck and Marcoulides ( 1996) 
associated various evaluative criteria with a number of types of evaluation: (a) standards- 
based evaluation which considers local context, job assignments, or particular priorities; 
(b) duties-based or job description evaluation; (c) outcome-based evaluation based 
primarily on behaviour and organizational outcomes; and (d) role-based evaluation 
considering best practice or action in a role over time in roles of governance, climate and 
social relations, and monitoring school instruction. 

In addition to performance effectiveness measures, other objects of evaluation 
have been identified as attitudes, cognitive processes, traits, and competence (Cullen, 
1995; Glasman & Heck (1992) cited in Heck & Marcouhdes, 1996). Braskamp, 
Brandenburg and Ory (1984) suggested that traits include not only instructor 
characteristics, but student and course characteristics as well. Other evaluative models 
support an even broader consideration of environmental traits to include culture, 
climate, or leadership (Heck & Marcoulides, 1996; Leithwood, 1994). 

In addition to major purposes for which performance evaluations are designed, 
other less direct outcome issues guide the development of criteria for appraisal systems. 
Because performance evaluations are interactive processes which involve multiple and 
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complex channels and methods of communication and feedback, standards must be 
estabhshed to protect of rights of parties to the evaluation in terms of confidentiahty and 
accoimtabihty. Standards must also address the utility of the evaluative process, that is 
whether it is informative, timely, and influential. The feasibiUty of the process relating 
to efficiency and pohtical viability are also critical in developing criteria (Genova, 

Madoff, Chin & Thomas, 1976; Heck & Marcouhdes, 1996). 

Perhaps the most important criteria for performance evaluations relates to 
accuracy of results (Heck & Marcouhdes, 1996). A number of measurement problems, 
however, often pose serious difficulties in this area. They include: (a) deficiency 
resulting from overlooking important aspects of actual performance; (b) unrehability 
resulting from poor quahty performance measures or disagreement among evaluators; 
and (c) invahdity resulting from a poorly-defined expectation of job performance (Nelson 
& Quick, 1994). These deficiencies in accuracy can often be related to inappropriate or 
insufficient numbers or types of evaluative criteria. 

In summary, based on a number of research approaches, it is clear that the 
criteria used to evaluate a number of performance measures are multiple and complex. 
They are not easily determined or explained by models which suggest straightforward 
relationships within a variety of education environments. Nor are they simple or static 
apphcations of those models within the context of dynamic teaching environments. 

Multiple sources of evaluative information. Information about 
instructors can be collected from multiple sources including faculty and department 
administrators, teaching peers, students, alumni, outside assessors, and self (Braskamp, 
Brandenburg & Ory, 1984; Braskamp & Ory, 1994; Roe & McDonald, 1983). Further, 
most researchers agree information from multiple sources is more rehable than that from 
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any single source. 

Evaluative information from multiple sources, however, requires a system of 
weighting or establishing the priority of sources. Just as various aspects of faculty work 
including research, professional service, citizenship, and teaching are weighted in the 
overall evaluation of faculty members, so are evaluative data from various sources as they 
relate to the teaching component (Genova, Madoff, Chin & Thomas, 1976). For example, 
faculty members may be evaluated based on a 40% research component, 10% 
professional service, 10% citizenship, with teaching accounting for 40% of the overall 
evaluation. Within teaching, however, information originating with a senior 
administrator may account for 25%, student evaluations 50%, and peer or self 
evaluations 25%. 

While information from multiple stakeholder groups contributes to the reliability 
of evaluative data, evidence suggests many stakeholders may not possess the requisite 
skills, sensitivity, authority, and training to perform credible performance evaluations 
(Cullen, 1995). To address potential inadequacies in criteria development and 
personnel skills in the evaluation system, self-evaluation is earning credibility as an 
innovative and effective instrument in generating both formative and summative 
performance information which provides a much broader range of evidence for effective 
instructional performance (Canadian Association of University Teachers, 1991). 

In summary, performance evidence provided by multiple sources enhances the 
fairness, usefulness, and credibility of the evaluation. In view of inadequacies within 
the evaluation system and among personnel involved, each source is not equally 
appropriate for all uses. However, multiple perspectives are deemed to support the 
overall value of assessment with each source providing a unique perspective. 
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Multiple methods of evaluation. The multiple perspectives approach to 

evaluation includes a variety of methods or techniques for collecting information which 

serves as indicators of performance in various areas. Several methods have been found 

to offer valuable information in appraising teaching. These include achievement tests, 

ratings and surveys, written essays or appraisal forms, interviews, personal observations, 

videotaping, and other specifically designed instruments tailored to particular 

circumstances. Other indicators of teaching eminence, quahty, and impact include 

prizes, awards, honours and invited addresses, and information from personal journals 

and pubhshed materials (Bosetti, 1996; Braskamp, Brandenburg & Ory, 1984; Braskamp 

& Ory, 1994; Canadian Association of University Teachers, 1991; Cullen, 1995; Genova, 

Madoff, Chin & Thomas, 1976; Roe & McDonald, 1983; Day, Robberecht & Roed, 1996). 

Self-Evaluation in Education Environments 

Braskamp & Ory (1994) suggest that reflective self-evaluation is the method by 

which instructors improve their teaching. 

Professors do not learn about how they function as professionals by first 
theorizing and then applying the theory to their work. Instead, they learn, 
understand, and change their work behavior by continuously examining, 
analyzing, hypothesizing, theorizing, and reflecting as they work. (p. 6) 

By viewing professors as learners, and learners as constructors of their own action 

and environment, we can understand teaching as something not learned from theory and 

text alone, but something based on experience and reflection as well (Bosetti, 1996; 

Braskamp & Ory, 1994; Culbertson, 1988; Griffiths, 1988; Jones, 1994). Self-evaluation 

offers instructors an opportunity to examine, analyze, and reflect on teaching strategies 

which, in response to environmental change, must also be adjusted and adapted on an 

ongoing basis. 

The necessity for ongoing reflective instructor evaluation means that assessment 
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traditionally performed by administrators is now frequently accompanied by self- 
evaluation. Self-evaluation allows instructors to better understand their world by 
reflecting on experiences, establishing goals and determining strategies for achieving 
these goals, monitoring their action, and continually re-evaluating and adjusting it. It is 
an on-going process which allows instructors to interpret unique challenges, develop 
innovative strategies, and record professional successes (Bosetti, 1996; Braskamp & Ory, 
1994; Johnson & Holdaway, 1991). 

Self-evaluation results have been criticized for a lack of objectivity in favoiu’ of the 
vested interests of the individual (Cullen, 1995; Genova, Madoff, Chin & Thomas, 1976). 
Other research, however, suggests 

faculty themselves are the most important assessment source because only 
they can provide descriptions of their work, the thinking behind it, and 
their own personal reporting, appraisals, interpretations, and goals. Self- 
assessment involves reflection and judgment. Only the professors 
themselves can make a case for their work. (Braskamp & Ory, 1994, p. 102) 

Further, self-ratings are shown to have good relative agreement with other ratings in 

teaching performance as demonstrated by similar profiles of strengths and weaknesses 

in instructor self-assessment and student ratings (Braskamp & Ory, 1994). 

Self-evaluations are increasingly used in performance evaluations, and some 

research suggests they result in more satisfying, constructive evaluations and less 

defensiveness concerning the evaluation process (DeGregorio & Fisher (1988) cited in 

Nelson & Quick, 1996). They not only benefit the individual being assessed, but self- 

evaluative data in the form of teaching records, portfohos, and dossiers also benefits the 

institute by promoting individual excellence and creative diversity (Bosetti, 1996; 

Braskamp & Ory, 1994; Canadian Association of University Teachers, 1991). 
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The Teaching Dossier: Multi-faceted Self-assessment Tool 

One method of self-evaluation is through a teaching dossier. The term dossier 
has been used interchangeably with a number of terms including portfolio, folio, and 
portrayal to represent a summary of an instructor's teaching activities. More 
specifically, a teaching dossier is a cumulative and coherent set of materials including 
work samples, reflective commentary, and selected short descriptions compiled by 
instructors to accurately convey the scope and quality of their teaching (Appendix 2) 
(Adamowicz, 1996; Barnett & Lee, 1994; Bosetti, 1996; Braskamp & Ory, 1994; Canadian 
Association of University Teachers, 1991; Day, Robberecht & Roed, 1996; Jones, 1994). 

Teaching dossiers can be distinguished from curricula vitarum by understanding 
teaching as one of four major parts of an instructor's role. While curriculum vitae 
records document a whole spectrum of faculty accomplishments relating to research, 
professional service, and citizenship, teaching dossiers record accomplishments 
specifically related to that major complex activity (Braskamp & Ory, 1994; Canadian 
Association of University Teachers, 1991). 

Multiple pixrposes of dossiers. Within the teaching component of faculty 
work there are two views of dossiers. While some researchers view them as a "posed 
photograph" representation of an instructor's best work or ideal performance, others 
utilize them as a 'candid snapshot' which reveals tough teaching challenges, issues, goals, 
and which reflects experimentation, failures, and successes (Adamowicz, 1996; Bosetti, 
1996; Braskamp & Ory, 1994; Canadian Association of University Teachers, 1991). 

As a result of the two perspectives of dossiers, views of their credibility are 
uneven (Bosetti, 1996; Braskamp & Ory, 1994). While some instructors regard them as 
accurate portrayals of teaching and learning (Barnett & Lee, 1994; Day, Robberecht & 
Roed, 1996; Jones, 1994), others believe they fail to capture individual approaches to 
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teaching and are too time-consuming to prepare (Adamowicz, 1996). For administrators, 
while some are uncomfortable making judgements of dossier quality, others find them 
more objective than evaluating teaching performance through the observation of a 
colleague (Braskamp & Ory, 1994). In summary, through a discussion of the two views of 
dossiers, an understanding of their value as formative and/or summative documents is 
seen as crucial to their credibility and acceptance in education environments. 

Multiple criteria within dossiers. Teaching dossiers offer advantages in 
providing evaluative data to the instructors themselves, administrators involved in the 
evaluation process, and the institute as a whole. They contribute to an instructor's 
improved teaching by stimulating self-reflection, self-analysis, and self-development. 
They also help instructors review their teaching philosophy, goals, and strategies and 
facihtate an instructor's active role in the evaluative process by including criteria which 
they feel are important (Adamowicz, 1996; Bosetti, 1996; Canadian Association of 
University Teachers, 1991; Day, Robberecht & Roed, 1996). 

Teaching dossiers also offer multiple advantages to evaluators and institutes. 

They offer richer, more authentic evidence of teaching effectiveness which reveals not 
only what occurred, but the rationale behind it as well. They provide evaluators an 
additional source of information, found often to coincide with student evaluations, upon 
which to consider the credibihty of multiple sources. Within the institute, a teaching 
dossier promotes collaboration and attention to shared goals by encouraging a view of 
teaching as scholarly activity through theorizing and apphcation of theory and personal 
philosophy, and reflection and rethinking the theory (Adamowicz, 1996; Braskamp & 

Ory, 1994; Canadian Association of University Teachers, 1991). 

Multiple methods of providing dossier information. Just as a self- 
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evaluation can comprise one component of a multi-faceted evaluation process, so can 
dossiers include a variety of documents useful for evaluation and development. Although 
dossier formats differ with every application and, likely, every teacher, typical 
characteristics relate to personal commitment, problem solving, and reflection (Jones, 
1994). 

Teaching dossiers can include descriptions of the type of teaching, for example, 
whether the presentation assumes a lecture format, occurs in a laboratory, or is as a 
thesis advisor. They include a reflective statement of teaching and learning goals and 
how they are accomplished, descriptions of activities undertaken to improve teaching, 
and contributions to the institution or profession. Supporting documentation including 
course development and instructional materials, documentation of student outcomes, 
evaluations of teaching, and honours or recognitions is also a useful component of 
teaching dossiers (Adamowicz, 1996; Day, Robberecht & Roed; 1996; Bosetti, 1996; 
Braskamp & Ory, 1994; Canadian Association of University Teachers, 1991). 

To summarize the multi-method approach to dossiers, the increasing use of such 
documentation as an evaluative tool in the education environment can be compared to 
the growing use of a variety of performance measures in student assessment (Braskamp 
& Ory, 1994). It can be argued that more collection of evidence for review and self- 
reflection is not just warranted, but requisite, in organizing instructors thinking about 
and achievements in teaching within dynamic environments. Because teaching dossiers 
can be used for many purposes, multiple sources and criteria of information are also 
crucial if the documents are to be accepted as credible. 

Conclusions and Implications 

This approach to performance evaluation emphasizes the importance of multiple 
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perspectives in appraising instructors within post-secondary environments. Information 
collected from a number of sources and through a variety of methods, each reflecting a 
diversity of purposes and criteria, support fair and credible assessment of instructor 
performance. It is therefore important that appropriate approaches are selected which 
meet the needs of individual instructors, administrators and institutes, and that such 
approaches reflect the dynamic contexts in which they occur. 

Although self-appraisal has been criticized as being biased, self-evaluation and 
reflection are important components of performance evaluation, both formative and 
summative, particularly when combined with other sources of evaluative data. Self- 
evaluation offers significant opportunity to address political, economic, and student 
changes which continue to transform today's post-secondary environments. Teaching 
dossiers offer a mechanism for educators to move beyond static conceptions of effective 
teaching in times of pohtical, cultural, and economic change. 

A number of imphcations must be considered if post-secondary institutes are to 
reap the benefits of evaluations programs which include dossiers. Professors must be 
encouraged to allocate the time and resources necessary to develop comprehensive and 
useful documentation. Evaluation systems must be established which assess the 
effectiveness of teaching dossiers, and administrators must be trained in their 
apphcation. Finally, professors and institutes must be recognized for outstanding efforts 
to implement dossier programs which improve the evaluation of teaching in current 
dynamic environments. 
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Appendix 1 



Multi-Faceted Evaluation Model 



MULTIPLE PURPOSES: 

* Formative (self-improve, curr) 

* Summative (promo, tenure, 
salary, student decisions) 



MULTIPLE CRITERIA: 

Teaching (Input, Process, Output) 
Traits (Self, Students, 

Environment) 



MULTI-FACETED PERFORMANCE EVALUATION 





CONTEXT RELATED: 

* Instructor 
* Student 

* Administration 

* Institution 

* Community 




Figure 1: Multi-faceted approach to instructor evaluation 

based on analysis by Braskamp, Brandenburg, & Ory (1984) and 
Genova, Madoff, Chin & Thomas (1976) 
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Appendix 2 



Teaching Dossier 

as Multi-faceted Evaluation Instrument 




PURPOSES: 
(self-improve, curr) 
(promo, tenure, 
salary, student decisions) 



lULTIPLE CRITERIA: 

* Teadhtog (Input, Process, Output)^ 

* Traitts (Self, Students, 

Environment) 





^ \ 





CONTEXT RELATED: 

* Instructor 
* Student 

* Administration 

* Institution 

* Community 




Figure 2: Teaching dossier as multi-faceted evaluative instrument 

based on Multi-faceted Evaluation Model (Appendix 1) 
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